Embedding constrained and unconstrained optimization programs as neural network layers

ABSTRACT

Aspects discussed herein may relate to methods and techniques for embedding constrained and unconstrained optimization programs as layers in a neural network architecture. Systems are provided that implement a method of solving a particular optimization problem by a neural network architecture. Prior systems required use of external software to pre-solve optimization programs so that previously determined parameters could be used as fixed input in the neural network architecture. Aspects described herein may transform the structure of common optimization problems/programs into forms suitable for use in a neural network. This transformation may be invertible, allowing the system to learn the solution to the optimization program using gradient descent techniques via backpropagation of errors through the neural network architecture. Thus these optimization layers may be solved via operation of the neural network itself.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority as a non-provisional of U.S.Provisional Patent Application No. 62/806,341, titled “EmbeddingConstrained and Unconstrained Optimization Programs as Neural NetworkLayers” and filed on Feb. 15, 2019, the disclosure of which isincorporated herein by reference in its entirety.

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF USE

Aspects of the disclosure relate generally to machine learning. Morespecifically, aspects of the disclosure may allow for the embedding ofconvex optimization programs in neural networks as a network layer, andmay allow the neural network to learn one or more parameters associatedwith the optimization program.

BACKGROUND

Neural networks and their constitutive layers often specify input-outputrelationships such that training procedures can readily identifyparameter values well-suited to given datasets and tasks. Often theserelationships are chosen to be analytic and differentiable to ensuregradient-based training methods are effective. Solving optimizationprograms in their original forms may be difficult for neural networks,particularly regarding those optimization programs with inputconstraints establishing permitted/feasible and/orunpermitted/infeasible values, because the constraints may cause thederivative and/or subdifferential of the optimization program functionto be ill-defined over the range of input that the neural network mayencounter.

Aspects described herein may address these and other problems, andgenerally improve the quality, efficiency, and speed of machine learningsystems. Further, aspects herein provide a practical application of thetransformations used to place optimization programs in suitable form forneural network processing.

SUMMARY

The following presents a simplified summary of various aspects describedherein. This summary is not an extensive overview, and is not intendedto identify key or critical elements or to delineate the scope of theclaims. The following summary merely presents some concepts in asimplified form as an introductory prelude to the more detaileddescription provided below.

Aspects of the disclosure relate to a procedure for mapping optimalityconditions associated with a broad class of optimization problemsincluding linear optimization programs and quadratic optimizationprograms, and more generally convex optimization programs, onto neuralnetwork structures using Cayley transforms. This may guarantee thatthese networks find solutions for convex problems.

Aspects discussed herein may relate to methods and techniques forembedding constrained and unconstrained optimization programs as layersin a neural network architecture. Systems are provided that implement amethod of solving a particular optimization problem by a neural networkarchitecture. Prior systems required use of external software topre-solve optimization programs so that previously determined parameterscould be used as fixed input in the neural network architecture. Aspectsdescribed herein may transform the structure of common optimizationproblems/programs into forms suitable for use in a neural network. Thistransformation may be invertible, allowing the system to learn thesolution to the optimization program using gradient descent techniquesvia backpropagation of errors through the neural network architecture.Thus these optimization layers may be solved via operation of the neuralnetwork itself. This may provide benefits such as improved predictionaccuracy, faster model training, and/or simplified model training, amongothers. Features described herein may find particular application withrespect to convex optimization programs, and may find particularapplication in recurrent neural network architectures and/orfeed-forward neural network architectures.

More particularly, some aspects described herein may provide acomputer-implemented method for embedding a convex optimization programas an optimization layer in a neural network architecture comprising aplurality of layers. According to some aspects, a computing systemimplementing the method may determine a set of parameters associatedwith the convex optimization program. The set of parameters may comprisea vector a=(a₁, a₂) of primal decision variables associated with theconvex optimization program and a vector b=(b₁, b₂) of dual decisionvariables associated with the convex optimization program. Vectors a andb may be related according to:

Aa ₁ =a ₂ and b ₁ =−A ^(T) b ₂

where A is a coefficient matrix corresponding to one or more constraintsof the convex optimization program, and A^(T) denotes the transpose ofmatrix A. The system may determine a set of intermediary functionalsƒ_(i)=(ƒ₁, ƒ₂). The value of ƒ_(i) may be defined as equal to a costterm associated with a corresponding a_(i) for a range of permittedvalues of a_(i) and equal to infinity for a range of unpermitted valuesof a_(i). The one or more constraints of the convex optimization programmay define the range of permitted values and the range of unpermittedvalues. The system may generate a set of network variables associatedwith the optimization layer based on applying a scattering coordinatetransform to vectors a and b. The set of network variables may comprisea vector c=(c₁, c₂) corresponding to input to the optimization layer anda vector d=(d₁, d₂) corresponding to an intermediate value of theoptimization layer.

The system may generate a linear component H of the optimization layerby applying a first transformation to coefficient matrix A, where thefirst transformation is of the form:

$H = {\begin{bmatrix}I & {- A^{T}} \\A & {I\mspace{34mu}}\end{bmatrix}\begin{bmatrix}{I\mspace{25mu}} & A^{T} \\{- A} & {I\mspace{20mu}}\end{bmatrix}}^{- 1}$

where I is the identity matrix. The linear component H may correspond toa linear mapping, linear operator, and/or weight matrix, and may be usedin the optimization layer to determine an intermediate value for dcorresponding to a given value for c.

The system may generate a non-linear component σ(·) of the optimizationlayer by applying a second transformation to the intermediaryfunctionals ƒ_(i), where the second transformation is of the form:

σ_(i)(d _(i))=(−1)^(i)(∂ƒ_(i) −I)(I+∂ƒ _(i))⁻¹(d _(i)),i=1,2,

where a ∂ƒ_(i) corresponds to the subdifferential of ƒ_(i). Thenon-linear component σ(⋅) may correspond to a non-linear mapping,non-linear operator, and/or non-linear transformation, and may be usedin the optimization layer to determine a next iteration value of c basedon application to a current iteration value for d.

The system may receive, by the optimization layer and from a prior layerof the neural network architecture, input values corresponding to vectorc. The system may iteratively compute, by the optimization layer, valuesfor vectors c and d to determine fixed point values c* and d*. Eachcomputation of a value for vectors c and d may be of the form:

d ^(n) =Hc ^(n) and c ^(n+1)=σ(d ^(n))

where n demotes the n-th iteration of the optimization layer, and c^(n)and d^(n) denote the n-th value for vectors c and d. The system maydetermine fixed point values a* and b* based on applying the inverse ofthe scattering coordinate transform to fixed point values c* and d*.

The system may provide, by the optimization layer, output based on fixedpoint values a* and b*. An error between a predicted output of theneural network architecture and an expected output for training dataused during a training process may be determined. The system maybackpropagate the determined error through the plurality of layers aspart of a machine learning process. The system may determine an updatedset of parameters associated with the convex optimization program basedon applying gradient descent to the linear component H and thenon-linear component σ(·). A trained model may be generated as a resultof repeated iterations of a machine learning process using the neuralnetwork architecture having the optimization layer described above. Thetrained model may be used to generate one or more predictions based on atrained set of parameters associated with the convex optimizationprogram.

Techniques described herein may flexibly be applied to any suitableneural network architecture. For example, techniques described hereinmay find application in neural network architectures such asconvolutional neural networks, recurrent neural networks, feed forwardneural networks, and the like, and combinations thereof. Similarly,techniques described herein may be applied to any suitable machinelearning application, such as speech recognition, image recognition, andothers.

Corresponding apparatus, systems, and computer-readable media are alsowithin the scope of the disclosure.

These features, along with many others, are discussed in greater detailbelow.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIG. 1 depicts an example of a computing device that may be used inimplementing one or more aspects of the disclosure in accordance withone or more illustrative aspects discussed herein;

FIG. 2 depicts an example neural network architecture for a modelaccording to one or more aspects of the disclosure;

FIGS. 3A, 3B, and 3C depict an exemplary optimization constraint andcorresponding subgradient relations according to one or more aspects ofthe disclosure;

FIG. 4 depicts a visual representation of generalized optimizationconstraints according to one or more aspects of the disclosure;

FIGS. 5A and 5B depict exemplary portions of a neural networkarchitecture according to one or more aspects of the disclosure; and

FIGS. 6A and 6B depict exemplary operations in a neural networkarchitecture to derive learned parameters according to one or moreaspects of the disclosure.

DETAILED DESCRIPTION

In the following description of the various embodiments, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration various embodiments in whichaspects of the disclosure may be practiced. It is to be understood thatother embodiments may be utilized and structural and functionalmodifications may be made without departing from the scope of thepresent disclosure. Aspects of the disclosure are capable of otherembodiments and of being practiced or being carried out in various ways.Also, it is to be understood that the phraseology and terminology usedherein are for the purpose of description and should not be regarded aslimiting. Rather, the phrases and terms used herein are to be giventheir broadest interpretation and meaning. The use of “including” and“comprising” and variations thereof is meant to encompass the itemslisted thereafter and equivalents thereof as well as additional itemsand equivalents thereof.

By way of introduction, aspects described herein may provide a methodfor designing neural networks that solve linear and quadratic programsduring inference using standard deep learning framework components.These networks are fully differentiable, allowing them to learnparameters for constraints and objective functions directly frombackpropagation, thereby enabling their use within larger end-to-endnetworks. Aspects of this disclosure are discussed generally withrespect to convex optimization programs. Illustrative examples ofstandard-form linear and quadratic optimization programs are discussed,as well as programs appearing in signal recovery and denoising contexts.

Before discussing these concepts in greater detail, however, severalexamples of a computing device that may be used in implementing and/orotherwise providing various aspects of the disclosure will first bediscussed with respect to FIG. 1.

FIG. 1 illustrates one example of a computing device 101 that may beused to implement one or more illustrative aspects discussed herein. Forexample, computing device 101 may, in some embodiments, implement one ormore aspects of the disclosure by reading and/or executing instructionsand performing one or more actions based on the instructions. In someembodiments, computing device 101 may represent, be incorporated in,and/or include various devices such as a desktop computer, a computerserver, a mobile device (e.g., a laptop computer, a tablet computer, asmart phone, any other types of mobile computing devices, and the like),and/or any other type of data processing device.

Computing device 101 may, in some embodiments, operate in a standaloneenvironment. In others, computing device 101 may operate in a networkedenvironment. As shown in FIG. 1, various network nodes 101, 105, 107,and 109 may be interconnected via a network 103, such as the Internet.Other networks may also or alternatively be used, including privateintranets, corporate networks, LANs, wireless networks, personalnetworks (PAN), and the like. Network 103 is for illustration purposesand may be replaced with fewer or additional computer networks. A localarea network (LAN) may have one or more of any known LAN topology andmay use one or more of a variety of different protocols, such asEthernet. Devices 101, 105, 107, 109 and other devices (not shown) maybe connected to one or more of the networks via twisted pair wires,coaxial cable, fiber optics, radio waves or other communication media.

As seen in FIG. 1, computing device 101 may include a processor 111, RAM113, ROM 115, network interface 117, input/output interfaces 119 (e.g.,keyboard, mouse, display, printer, etc.), and memory 121. Processor 111may include one or more computer processing units (CPUs), graphicalprocessing units (GPUs), and/or other processing units such as aprocessor adapted to perform computations associated with machinelearning. I/O 119 may include a variety of interface units and drivesfor reading, writing, displaying, and/or printing data or files. I/O 119may be coupled with a display such as display 120. Memory 121 may storesoftware for configuring computing device 101 into a special purposecomputing device in order to perform one or more of the variousfunctions discussed herein. Memory 121 may store operating systemsoftware 123 for controlling overall operation of computing device 101,control logic 125 for instructing computing device 101 to performaspects discussed herein, machine learning software 127, training setdata 129, and other applications 129. Control logic 125 may beincorporated in and may be a part of machine learning software 127. Inother embodiments, computing device 101 may include two or more of anyand/or all of these components (e.g., two or more processors, two ormore memories, etc.) and/or other components and/or subsystems notillustrated here.

Devices 105, 107, 109 may have similar or different architecture asdescribed with respect to computing device 101. Those of skill in theart will appreciate that the functionality of computing device 101 (ordevice 105, 107, 109) as described herein may be spread across multipledata processing devices, for example, to distribute processing loadacross multiple computers, to segregate transactions based on geographiclocation, user access level, quality of service (QoS), etc. For example,devices 101, 105, 107, 109, and others may operate in concert to provideparallel computing features in support of the operation of control logic125 and/or software 127.

One or more aspects discussed herein may be embodied in computer-usableor readable data and/or computer-executable instructions, such as in oneor more program modules, executed by one or more computers or otherdevices as described herein. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data typeswhen executed by a processor in a computer or other device. The modulesmay be written in a source code programming language that issubsequently compiled for execution, or may be written in a scriptinglanguage such as (but not limited to) HTML or XML. The computerexecutable instructions may be stored on a computer readable medium suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects discussed herein, and such data structuresare contemplated within the scope of computer executable instructionsand computer-usable data described herein. Various aspects discussedherein may be embodied as a method, a computing device, a dataprocessing system, or a computer program product.

Having discussed several examples of computing devices which may be usedto implement some aspects as discussed further below, discussion willnow turn to a method for embedding convex optimization programs inneural network layers.

FIG. 2 illustrates an example neural network architecture 200. Anartificial neural network may be a collection of connected nodes, withthe nodes and connections each having assigned weights used to generatepredictions. Each node in the artificial neural network may receiveinput and generate an output signal. The output of a node in theartificial neural network may be a function of its inputs and theweights associated with the edges. Ultimately, the trained model may beprovided with input beyond the training set and used to generatepredictions regarding the likely results. Artificial neural networks mayhave many applications, including object classification, imagerecognition, speech recognition, natural language processing, textrecognition, regression analysis, behavior modeling, and others.

An artificial neural network may have an input layer 210, one or morehidden layers 220, and an output layer 230. Illustrated networkarchitecture 200 is depicted with three hidden layers. The number ofhidden layers employed in neural network 200 may vary based on theparticular application and/or problem domain. For example, a networkmodel used for image recognition may have a different number of hiddenlayers than a network used for speech recognition. Similarly, the numberof input and/or output nodes may vary based on the application. Manytypes of neural networks are used in practice, such as convolutionalneural networks, recurrent neural networks, feed forward neuralnetworks, combinations thereof, and others. Aspects described herein maybe used with any type of neural network, and for any suitableapplication.

During the model training process, the weights of each connection and/ornode may be adjusted in a learning process as the model adapts togenerate more accurate predictions on a training set. The weightsassigned to each connection and/or node may be referred to as the modelparameters. The model may be initialized with a random or white noiseset of initial model parameters. The model parameters may then beiteratively adjusted using, for example, gradient descent algorithmsthat seek to minimize errors in the model.

A problem domain to be solved by the neural network may include anassociated optimization program. The optimization program may beconstrained and/or unconstrained, and may assign weights and/orpenalties to certain input values and/or combinations of input values.The weights (e.g., coefficient matrices) and/or constraints (e.g.,maximum values or relations between different inputs) may be referred toas parameters of the optimization program. Solving for these parametersmay be necessary to generate suitable predictions in certain problemdomains. Convex optimization programs are a class of optimizationprograms that are commonly encountered in machine learning applications,and aspects herein may find particular application with respect toconvex optimization programs. This disclosure discusses the example ofinput-output relationships characterized by standard-form linear andquadratic optimization programs. Aspects of this disclosure may findapplication more generally in the context of convex optimizationprograms.

Linear optimization programs may be those written according to equation(LP):

$\begin{matrix}\min\limits_{x} & {q^{T}x} \\{s.t.} & {{Ax} \leq r} \\\; & {x \geq 0}\end{matrix}$

While quadratic optimization programs may be those written according toequation (QP):

$\begin{matrix}\min\limits_{x} & {{\frac{1}{2}x^{T}{Qx}} + {q^{T}x}} \\{s.t.} & {{Ax} \geq r}\end{matrix}$

where x is a vector of primal decision variables, q is a cost vector, ris an inequality vector, and A and Q are respectively linear andquadratic coefficient matrices.

Solving optimization programs in their original forms may be difficultfor neural networks, particularly regarding those optimization programswith input constraints establishing permitted/feasible and/orunpermitted/infeasible values, because the constraints may cause thederivative and/or subdifferential of the optimization program functionto be ill-defined over the range of input that the neural network mayencounter. For example, FIG. 3A depicts an illustrative function ƒ(a)300 that may provide a cost value associated with input vector a. Asillustrated, permissible values of vector a may be bounded by variablesl and u. Function ƒ(a) 300 may indicate that the cost value is 0 for allpermissible values, and infinite otherwise. Applying canonicalsubgradient approaches to ƒ(a) 300 may not be able to locate a minima byfollowing the gradient descent based on ∂ƒ(d), illustrated in FIG. 3Bchart 310 with exemplary constraints l=−2 and u=2. Aspects describedherein may apply a transformation to the cost function and parameters ofthe optimization program to derive a fully differentiable representationof the cost function as described further below, illustrated in FIG. 3Cchart 320 with the same exemplary constraints.

Aspects described herein introduce an optimization layer, such asoptimization layer 225 illustrated in FIG. 2, which may be configuredsuch that the optimization program is represented in a form suitable forsolving by the processing of the neural network itself. Structures ofthe optimization layer are illustrated in FIGS. 5A and 5B. Theoptimization layer may include a linear component and a non-linearcomponent generated by applying the transformations discussed furtherherein to the parameters of the optimization program, and the componentsmay operate on network variable transformations of the set of parametersfor the optimization program. Further details regarding thetransformation of the optimization program to the network coordinatesystem, the generation of the linear and non-linear components of theoptimization layer, and the structure and operation of the optimizationlayer are discussed further below.

Aspects described herein may present a method for solving a particularoptimization program by a neural network. Broadly speaking, the systemmay transform the problem into a neural network structure that can beembedded and trained inside of a larger end-to-end network. An instanceof the optimization problem may be solved by executing the network untila fixed-point is identified or another appropriate threshold for what isconsidered a fixed-point is met. The system may transform thefixed-point back to the original coordinate system, thereby obtaining atrained value relevant to the optimization program.

In particular, the system may transform the problem into a neuralnetwork structure that can be embedded and trained inside of a largerend-to-end network by transforming the problem parameters in aninvertible and differentiable matter. The problem may be transformedinto a direct and/or residual neural network form. A direct formstructure may comprise a single matrix multiplication followed by anon-linearity, as illustrated by network structure 500 in FIG. 5A. Aresidual form structure may comprise the matrix multiplication andnon-linearity, but may also have an identity path from the input and aweighted combination of the input and the product of the multiplicationand non-linearity, as illustrated by network structure 550 in FIG. 5B.Both the direct and residual forms may make use of standard neuralnetwork components, such as fully connected layers, matrixmultiplication, and non-linearities such as rectified linear units(ReLUs).

The system may solve an instance of the optimization problem byexecuting the optimization layer of the network until a fixed-point isidentified or an appropriate threshold is met. This may be done in thedirect and/or residual forms. The variables operating on in the solvingmay be specific combinations of primal and dual decision variablesrather than the primal or dual variables themselves. The optimizationproblem parameters can be fixed, pre-defined, and/or learned from dataand/or some combination of fixed and learned. The application of thetransformations described herein to neural network processing may enablethe embedding of an optimization layer allowing the neural network tosolve an associated optimization program. The optimization problem maybe solved directly in the inference and/or forward pass of the networkwithout requiring the use of external optimization tools.

This procedure may be performed on every forward pass of the network aspart of larger network training. The optimization problem parameters maybe updated in the same manner that other parameters of the largernetwork are learned. These features may also allow any neural networkinference framework, such as TensorRT from NVIDIA CORP, to be used as aconstrained and unconstrained optimization solver. The optimizationlayer may be configured to run in a recurrent loop to arrive at thefixed-point. In a given iteration of the end-to-end neural network,multiple iterations of the optimization layer may run as a result of therecurrent (and or similar feed forward) structure. The fixed-pointresults serve as the network variables for the optimization layer forthat iteration of the end-to-end neural network, and are updated in thesame manner as other parameters, e.g., through gradient descent.

Optimality Conditions

To begin, let a=(a₁, a₂) and b=(b₁, b₂) respectively denote vectors ofprimal and dual decision variables associated with an optimizationprogram. The linear relationships imposed on the decision variables areenforced according to equation (1):

Aa ₁ =a ₂ and b ₁ =−A ^(T) b ₂

where A is a coefficient matrix associated with the constraints of theoptimization program, and where A^(T) denotes the transpose of matrix A.

Equivalently, the linear feasibility constraints imposed on the decisionvariables a and b above correspond to the behavioral statement shown inequation (2):

$\begin{bmatrix}a_{1} \\b_{1} \\a_{2} \\b_{2}\end{bmatrix} \in {{range}\mspace{14mu} {\left( \begin{bmatrix}I & {0\mspace{34mu}} \\0 & {- A^{T}} \\A & {0\mspace{34mu}} \\0 & {I\mspace{34mu}}\end{bmatrix} \right).}}$

where I is the identity matrix. Moving forward, the vector subspace inequation (2) is denoted as

.

Next, the remainder of the optimality conditions associated with costterms and inequality constraints are enforced using a set

describing admissible configurations of the decision vector (a, b).

may be decomposed according to

=

₁×

₂ where each

_(i) is a set relation restricting the variables (a_(i), b_(i)) fori=1,2. To generate

_(i), an intermediary functional ƒ_(i) may be defined equal to the costterm associated with a_(i) over its feasible domain and equal toinfinity for infeasible values. For example, the primal vector a_(i) inequation (LP) is restricted to be elementwise non-negative with costterm q^(T)a_(i), therefore the functional ƒ₁ is shown in equation (3):

${f_{1}\left( a_{1} \right)} = \left\{ \begin{matrix}{{q^{T}a_{1}},} & {{a_{1} \geq 0}\mspace{31mu}} \\{{\infty,}\mspace{25mu}} & {otherwise}\end{matrix} \right.$

The set

_(i) is then the set of values (a_(i),b_(i)) where b_(i)=∂ƒ₁ (a₁) is thesubdifferential of ƒ_(i). A complete description of solutions to theoptimality conditions is then the set of elements (a, b) in

∩

. FIG. 4 illustrates these optimality conditions graphically in visualrepresentation 400.

Scattering Coordinate Transformations

For non-smooth convex cost functions over general convex sets, the(a_(i),b_(i)) relationship encapsulated by

_(i) is not necessarily functional, thus inserting state into thestructure in FIG. 1(a) may not yield well-defined iterations or loops.To circumvent this difficulty, aspects may proceed by mapping theoptimality conditions onto the neural network structures in FIGS. 5A and5B. FIG. 5A depicts a direct network structure 500 comprising a linearcomponent H 510 and non-linear component σ(·) 520. Linear component H510 may be a linear mapping, linear operator, and/or weight matrixapplied to input network variable c. Non-linear component σ(·) 520 maybe a non-linear mapping, non-linear operator, and/or non-lineartransformation applied to intermediate network variable d. FIG. 5Bdepicts an residual network structure that also includes linearcomponent H 510 and non-linear component σ(·) 520.

Toward this end, network variables c=(c₁, c₂) and d=(d₁, d₂) may beproduced from the decision variables a and b according to the scatteringcoordinate transform shown in equation (4):

$\begin{bmatrix}c_{1} \\d_{1} \\c_{2} \\d_{2}\end{bmatrix} = {\underset{\underset{\overset{\Delta}{=}M}{}}{\begin{bmatrix}I & {- I} & {0\mspace{14mu}} & 0 \\I & {I\mspace{14mu}} & {0\mspace{14mu}} & 0 \\0 & {0\mspace{14mu}} & {- I} & I \\0 & {0\mspace{14mu}} & {I\mspace{14mu}} & I\end{bmatrix}}\begin{bmatrix}a_{1} \\b_{1} \\a_{2} \\b_{2}\end{bmatrix}}$

Next, this transform may be reflected onto the optimality conditions byproviding analytic expressions for the network parameter matrices H andactivation functions σ(·) which encapsulate the transformed optimalityconditions M(

∩

) as well as address issues of their well-posedness. The neural networkscan then be unrolled or iterated until fixed-points (c*, d*) areidentified which in turn can be used to identify solutions (a*, b*) inthe original coordinate system by inverting equation (4).

To assist with the strategy above, note that the transformed behavior M

is the vector space described by equation (5):

$\begin{bmatrix}c_{1} \\d_{1} \\c_{2} \\d_{2}\end{bmatrix} \in {{range}\mspace{14mu} {\left( \begin{bmatrix}{I\mspace{25mu}} & {A^{T}\mspace{14mu}} \\{I\mspace{25mu}} & {- A^{T}} \\{- A} & {I\mspace{34mu}} \\{I\mspace{25mu}} & {I\mspace{34mu}}\end{bmatrix} \right).}}$

Consistent with the illustration in FIG. 5A, aspects may proceed bymapping the optimization problem to the network structures with theconvention that c and d are matrix inputs and outputs, respectively.Therefore, the behavior in equation (5) reduces to the matrix equationshown in equation (6):

$\begin{bmatrix}d_{1} \\d_{2}\end{bmatrix} = {{\underset{\underset{\overset{\Delta \;}{=}H}{}}{{\begin{bmatrix}{I\;} & {- A^{T}} \\A & {I\mspace{34mu}}\end{bmatrix}\begin{bmatrix}{I\mspace{25mu}} & A^{T} \\{- A} & {I\mspace{20mu}}\end{bmatrix}}^{- 1}}\begin{bmatrix}c_{1} \\c_{2}\end{bmatrix}}.}$

This form of H corresponds to the Cayley transform of a skew-symmetricmatrix characterizing the dual extension of the matrix A, thus it isboth orthogonal and well-defined for arbitrary A. Efficient methods forcomputing H follow from standard algebraic reduction mechanisms such asthe matrix inversion lemma.

Concerning the transformed relation M

, convexity of the functionals ƒ_(i) paired with the coordinatetransform in equation (4) provides that c_(i) can always be written as afunction of d_(i) for i=1,2. To see this, let σ_(i) denote the mappingand note that the subgradient relation in

_(i) can be transformed and written according to equation (7):

σ_(i)(d _(i))=(−1)^(i)(∂ƒ_(i) −I)(I+∂ƒ _(i))⁻¹(d _(i)),i=1,2,

where σ_(i) is closely related to the Cayley transform of a ∂ƒ_(i).Drawing upon this observation, a well-known result in convex analysisstates that since a ∂ƒ_(i) is the subdifferential of a convex functionit is monotone, therefore I+∂ƒ_(i) is strongly monotone since it is thesum of a monotone and strongly monotone function. Invertibility of theterm I+∂ƒ_(i), and thus the validity of equation (7) as a well-definedoperator, then follows from application of the theorem of Browder-Mintyon monotone operators. Moreover, it follows that the σ_(i) is passive ornon-expansive, i.e. is Lipschitz continuous with constant not greaterthan unity.

Convergence Analysis

This disclosure will next show that the iterating the residual structurein FIG. 5C yields solutions to the transformed optimality conditions M(

∩

) associated with convex problems such as (LP) and (QP). Moreconcretely, it will be shown that the sequence shown in equation (8):

c ^(n+1) =ρc ^(n)+ρ ^(T)(c ^(n)),n∈

tends to a fixed-point c*=T(c*) for ρ∈(0,1) and where ρ=1−ρ. Once c* isobtained, it is straightforward to produce d* using equation (5) fromwhich solutions (a*, b*) can be obtained by inverting equation (4).

For general convex problems, T is non-expansive since it is thecomposition of an orthogonal matrix H and the non-expansive operator ain equation (7). Therefore, a single iteration of equation (8) yieldsequation (8.1):

∥c ^(n+1) −c*∥ ² =ρ∥c ^(n) −c*∥ ² +ρ∥T(c ^(n))−c*∥ ² −ρρ∥T(c ^(n))−c^(n)∥² ≤∥c ^(n) −c*∥ ² −ρρ∥T(c ^(n))−c ^(n)∥²

where the equality is due to the application of Stewarts Theorem and theinequality is due to both the non-expansivity and fixed-point propertiesof T. Iterating the inequality yields equation (9):

${{c^{n + 1} - c^{*}}}^{2} \leq {{{c^{0} - c^{*}}}^{2} - {\rho \overset{\_}{\rho}{\sum\limits_{l = 0}^{n}\; {{{{T\left( c^{l} \right)} - c^{l}}}^{2}.}}}}$

Loosening equation (9) further and taking a limit provides the boundshown in equation (10):

${\lim\limits_{n\rightarrow\infty}{\sum\limits_{n}{{{T\left( c^{n} \right)} - c^{n}}}^{2}}} \leq {\frac{1}{\rho \overset{\_}{\rho}}{{{c^{0} - c^{*}}}^{2}.}}$

Since equation (10) is bounded above it follows that T(c^(n))→c^(n) andtherefore c^(n)→c* which concludes the argument of convergence forgeneral convex optimization problems.

For the important special case of strictly convex cost functions overconvex sets, the nonlinearity in equation (7) reduces to a contractivemapping. Consequently, T is also contractive and the process ofiterating either the direct or residual network structures results inlinear convergence to a solution or fixed-point. A proof of this factfollows from direct application of the Banach fixed-point theorem.

Connections to Proximal Algorithms

Proximal iterations for minimizing non-differentiable, convex functionsƒ:

^(N)→

∪{∞} take the general form shown in equation (11):

x ^(n)=prox_(ρƒ)(x ^(n−1)),n∈

where ρ is a tuning parameter and the scaled proximal operatorprox_(ρƒ):

^(N)→

^(N) is defined according to equation (12):

${{prox}_{\rho \; f}(x)}\overset{\Delta}{=}{{\arg {\min\limits_{z}{f(z)}}} + {\frac{1}{2\rho}{{{x - z}}^{2}.}}}$

To connect the scattering algorithms outlined in this section with theirproximal counterparts, it is shown that the proximal operator inequation (12) corresponds to a different transformation of

_(i) associated with ƒ. In particular, the scaled proximal operator isrelated to the subgradient ∂ƒ according to equation (13):

$\begin{bmatrix}{{prox}_{\rho \; f}(d)} \\{d}\end{bmatrix} = {\begin{bmatrix}I & 0 \\I & {\rho \; I}\end{bmatrix}\begin{bmatrix}{x\mspace{50mu}} \\{\partial{f(x)}}\end{bmatrix}}$

To prove this relationship holds it must be shown that d=(I+ρ∂ƒ)(prox_(ρƒ)(d)). To do this, let v*=prox_(ρƒ)(d) and consider thefunction

${p(v)} = {{f(v)} + {\frac{1}{2\rho}{{{v - d}}^{2}.}}}$

The condition ∂p(v*)=0 then yields the constraint ρ∂ƒ(v*)+v*=d which isprecisely the relationship in equation (13). Therefore, proximal andscattering methods are related through different coordinatetransformation matrices M.

Example Network Modules

In the following sections, this disclosure explains the derivation ofthe requisite nonlinearities to build networks that themselves solvespecific forms of linear and quadratic programming problems. The forwardpass of a residual network is portrayed in FIGS. 6A and 6B, wherein theportion in FIG. 6A is implemented iteratively until a fixed-point isidentified from which the portion in FIG. 6B then returns the primal anddual solutions. FIG. 6B illustrates the inverse scattering transformused on fixed-points (c*, d*). Transformations from the parameters A, q,r and Q to produce H and a may only be required for parameters that areeither network inputs or to be learned and only done once upon networkinitialization for those that are static.

Linear Optimization Program Network Modules

The optimality conditions for linear programming problems written instandard form directly map to FIG. 4 with decision variables (a₁,a₂)=(x, Ax). The relation

₁ is generated using the example functional in equation (3) and

₂ uses the indicator functional ƒ₂ given by equation (14)

${f_{2}\left( a_{2} \right)} = \left\{ \begin{matrix}{{0,}\mspace{11mu}} & {{a_{2} \leq r}\mspace{34mu}} \\{\infty,} & {otherwise}\end{matrix} \right.$

The nonlinearities for linear optimization program (LP) modules aregenerated using equation (7) and implemented coordinatewise using theexpressions shown in equations (15) and (16):

σ₁(d ₁)=|d ₁ −q|−q

σ₂(d ₂)=|d ₂ −r|−r

which are both easily verified to be non-expansive and can be formedusing compositions of standard ReLU activations.

Quadratic Optimization Program Network Modules

The optimality conditions for quadratic programming problems written instandard form directly map to FIG. 4 with decision variables (a₁,a₂)=(x, Ax). The relations

_(i) correspond to the functionals shown in equations (17) and (18):

${f_{1}\left( a_{1} \right)} = {{\frac{1}{2}a_{1}^{T}{Qa}_{1}} + {q^{T}a_{1}}}$${f_{2}\left( a_{2} \right)} = \left\{ \begin{matrix}{{0,}\mspace{11mu}} & {{a_{2} \geq r}\mspace{34mu}} \\{\infty,} & {otherwise}\end{matrix} \right.$

The nonlinearities for quadratic optimization program (QP) modules aregenerated using equation (7) and implemented using the expressions shownin equations (19) and (20):

σ₁(d ₁)=(I−Q)(I+Q)⁻¹(d ₁ −q)−q

σ₂(d ₂)=|d ₂ −r|−r

It is straightforward to show that σ₁ is contractive if Q is positivedefinite, non-expansive if Q is positive semidefinite, and expansive ifQ is indefinite and σ₂ is coordinatewise non-expansive.

Sparse Signal Recovery Network Modules

In a variety of applications, nonlinear features naturally arise inprograms that reduce to (LP) or (QP). Recasting tricks have beendeveloped in response to this. These same tricks may be combined intandem with the networks in this disclosure. Additionally and/oralternatively, specialized networks can be assembled by designingnon-linearities that directly represent the nonlinear features using theprocedures described herein.

For example, generating sparse solutions to underdetermined linearsystems of equations satisfying certain spectral properties is a linearprogram often cast as the Basis Pursuit (BP) problem shown in equation(21):

${\min\limits_{x}\mspace{14mu} {{x}_{1}\mspace{14mu} {s.t.\mspace{14mu} {Ax}}}} = r$

Moreover, equation (21) has been extended via regularization to handlecases where the measurement vector r contains noise and Ax is onlyrequired to be reasonably close to r. This recovery problem is aquadratic program often cast as the Basis Pursuit Denoising problem(BPDN) according to equation (22):

${{\min\limits_{x}\mspace{14mu} {\lambda {x}_{1}}} + {\frac{1}{2}{{v - r}}_{2}^{2}\mspace{14mu} {s.t.\mspace{14mu} {Ax}}}} = v$

where λ balances the absolute size of the solution with the desiredagreement of Ax and r. Rather than recasting into standard form byintroducing auxiliary variables and additional constraints, aspects maynext define the network modules directly from the objective functions.The optimality conditions directly map to FIG. 4 where (a₁, a₂)=(x, Ax)and the relations

_(i) produce the nonlinearities shown below in equations (23), (24), and(25):

${{For}\mspace{14mu} (21)\mspace{14mu} {and}\mspace{14mu} (22)\mspace{14mu} {\sigma_{1}\left( d_{1} \right)}} = \left( {{\begin{matrix}{{{- d_{1}},}\mspace{115mu}} & {{d_{1}} \leq \lambda} \\{{d_{1} - {2\lambda \; {{sgn}\left( d_{1} \right)}}},} & {{d_{1}} > \lambda}\end{matrix}{For}\mspace{14mu} (21)\mspace{14mu} {\sigma_{2}\left( d_{2} \right)}} = {{d_{2} - {2r{For}\mspace{14mu} (22)\mspace{14mu} {\sigma_{2}\left( d_{2} \right)}}} = {- r}}} \right.$

The activations in equations (23) and (24) are non-expansive and theactivation in equation (25) is contractive with Lipschitz constant equalto zero.

Total Variation Denoising Network Modules

Similar to the recovery of sparse signals, the denoising of certainsignal models from noisy measurements is often cast using quadraticprograms. As a concrete example of this, the total variation denoisingproblem attempts to denoise or smooth observed signals y using anapproximation x produced according to equation (26):

${{\min\limits_{x}{\frac{1}{2}{{x - y}}_{2}^{2}}} + {\lambda {v}_{1}\mspace{14mu} {s.t.\mspace{14mu} {Dx}}}} = v$

where the form of the parameter matrix D encodes the targeted signalmodel and 2 balances the approximation and model penalties. When D takesthe form of a first-order difference operator, i.e. with rowse_(i)−e_(i+1), the penalty ∥Dx∥₁ encourages the signal to tend toward apiece-wise constant construction.

The optimality conditions associated with equation (26) directly map toFIG. 4 where (a₁, a₂)=(x, v) and the relations

_(i) produce the nonlinearities shown in equations (27) and (28):

σ₁(d₁) = y ${\sigma_{2}\left( d_{2} \right)} = \left( \begin{matrix}{{d_{2},}\mspace{155mu}} & {{d_{2}} \leq \lambda} \\{{{2\lambda \mspace{20mu} {{sign}\left( d_{2} \right)}} - d_{2}},} & {{d_{2}} > \lambda}\end{matrix} \right.$

which are both easily verified to be non-expansive. Observe thatequation (28) is the negative of equation (23) consistent with theformulation in equation (7) and the fact that x maps to a₁ in equation(22) and v maps to a₂ in equation (26).

Example Application—Learning a in BP and BPDN

As one example of an application of some of the processes and proceduresdiscussed herein, the system may be configured to learn constraints inlinear and quadratic programs from data by learning the measurementmatrix A for the BP problem of equation (21) and BPDN problem ofequation (22). A dataset may be constructed by randomly drawing an M×Nmatrix A* and producing network input-target samples (r, x*) by randomlybuilding K-sparse vectors x* and computing the companion measurementsr=A*x+z where K is drawn uniformly over the interval [K_(min), K_(max)]and z is a noise vector with entries sampled from

(0, σ²). The neural networks take measurements r as inputs and produceoutputs x which solve BP or BPDN for the current parameter matrix Aduring the forward pass. The training objective is to minimize∥{circumflex over (x)}−x*∥₁, i.e. to resolve the difference between thesparse vector x* which produces r using A* and the network output.

The networks may be trained using stochastic gradient descent with alearning rate of 0.01, a batch size of 64 and no momentum or weightdecay terms. The training and validation splits in this exampleimplementation comprise 256,000 and 10,000 samples, respectively.

Consistent with training neural networks via non-convex optimization,meaning gradient-based methods generally find local minima, severaltrials have been observed that disparate matrices A produce similarvalidation errors whereas the validation error at the global minima A*is slightly lower. This observation corroborates the fact that traininglinear and quadratic program parameters is also a non-convexoptimization problem. Warm starting from A* corrupted by noise andfinetuning repeatedly yielded A* as a solution.

Example Application—Learning D in TVDN Problems

As another example of an application of some of the processes andprocedures discussed herein, the system may be configured to learnparameters in a signal processing algorithm from data by learning thedenoising matrix D in equation (26). A dataset may be constructed bygenerating input-target samples (y, x*) of piecewise constant signals x*and their noise corrupted companions y. The neural network takes y asinput and produces output {circumflex over (x)} which solves TVDN forthe current denoising matrix D during the forward pass. The trainingobjective is to minimize ∥{circumflex over (x)}−x*∥₁, i.e. to resolvethe difference between the piecewise constant signal x* and the networkoutput.

The examples in this section primarily serve to underscore thefeasibility and numerical stability of unrolling constrainedoptimization algorithms within the deep learning paradigm, andsecondarily as an interesting class of algorithms themselves. Theability to learn convex programs as essentially layers within a largernetwork, similar to learning affine or convolution layers, may enablemany new neural network architectures and applications that can takeadvantage of such architectures.

In accordance with the above detailed description, aspects describedherein may provide a computer-implemented method for embedding a convexoptimization program as an optimization layer in a neural networkarchitecture. Exemplary steps of such a method 700 are shown in FIG. 7.

At step 705, a computing device may determine a set of parametersassociated with the convex optimization program. The set of parametersmay include a vector of primal decision variables and a set of dualdecision variables. A coefficient matrix may also be determined, whichmay correspond to one or more constraints of the convex optimizationprogram.

At step 710, the computing device may determine a set of intermediaryfunctions having values defined as equal to a cost term associated withcorresponding values of the primal decision variables. The one or moreconstraints of the convex optimization program may define a range ofpermitted values and a range of unpermitted values. The cost term may beequal to infinity for unpermitted values of the primal decisionvariables and equal to other, non-infinite values for permitted valuesof the primal decision variables.

At step 715, the computing device may generate a set of networkvariables associated with the optimization layer based on applying ascattering coordinate transform to the vector of primal decisionvariables and the vector of dual decision variables. The networkvariables may comprise a vector corresponding to inputs to theoptimization layer, and a vector corresponding to intermediate values ofthe optimization layer.

At step 720, the computing device may generate a linear component of theoptimization layer by applying a transformation to the coefficientmatrix.

At step 725, the computing device may generate a non-linear component ofthe optimization layer by applying a transformation to the intermediaryfunctionals.

At step 730, the computing device may operate a neural network includingthe optimization layer, comprising the generated linear component andthe non-linear component. The computing device may receive, by theoptimization layer and from a prior layer of the neural network, inputvalues corresponding to the inputs to the optimization layer. Thecomputing device may iteratively compute, by the optimization layer,values for the network variables to determine fixed point values for thenetwork variables.

At step 735, the computing device may determine fixed point values forthe primal decision variables and the dual decision variables based onapplying the inverse of the scattering coordinate transform to the fixedpoint values for the network variables.

At step 740, the computing device may provide, by the optimizationlayer, first output based on the determined fixed point values for theprimal decision variables and the dual decision variables to a nextlayer of the neural network.

At step 745, the computing device may determine an error based on secondoutput of the neural network, wherein the second output is based on thefirst output of the optimization layer. The second output may be anoutput of a last layer of the neural network.

At step 750, the computing device may backpropagate the determined errorthrough the plurality of layers of the neural network. Thebackpropagating may comprise determining an updated set of parametersassociated with the convex optimization program based on applyinggradient descent to the linear component and the non-linear component.

At step 755, the computing device may generate one or more predictionsby the neural network based on a trained set of parameters associatedwith the convex optimization program.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. A computer-implemented method for embedding aconvex optimization program as an optimization layer in a neural networkarchitecture comprising a plurality of layers, the method comprising:determining a set of parameters associated with the convex optimizationprogram, wherein the set of parameters comprises a vector a=(a₁, a₂) ofprimal decision variables associated with the convex optimizationprogram and a vector b=(b₁, b₂) of dual decision variables associatedwith the convex optimization program, where:Aa ₁ =a ₂ and b ₁ =−A ^(T) b ₂ and where A is a coefficient matrixcorresponding to one or more constraints of the convex optimizationprogram, and A^(T) denotes the transpose of matrix A; determining a setof intermediary functionals ƒ_(i)=(ƒ₁, ƒ₂), wherein the value of ƒ_(i)is defined as equal to a cost term associated with a corresponding a_(i)for a range of permitted values of a_(i) and equal to infinity for arange of unpermitted values of a_(i), wherein the one or moreconstraints of the convex optimization program define the range ofpermitted values and the range of unpermitted values; generating a setof network variables associated with the optimization layer based onapplying a scattering coordinate transform to vectors a and b, whereinthe set of network variables comprises a vector c=(c₁, c₂) correspondingto input to the optimization layer and a vector d=(d₁, d₂) correspondingto an intermediate value of the optimization layer; generating a linearcomponent H of the optimization layer by applying a first transformationto coefficient matrix A, where the first transformation is of the form:$H = {\begin{bmatrix}I & {- A^{T}} \\A & {I\mspace{34mu}}\end{bmatrix}\begin{bmatrix}{I\mspace{25mu}} & A^{T} \\{- A} & {I\mspace{20mu}}\end{bmatrix}}^{- 1}$ where I is the identity matrix; generating anon-linear component σ(·) of the optimization layer by applying a secondtransformation to the intermediary functionals ƒ_(i), where the secondtransformation is of the form:σ_(i)(d _(i))=(−1)^(i)(∂ƒ_(i) −I)(I+∂ƒ _(i))⁻¹(d _(i)),i=1,2, where∂ƒ_(i) corresponds to the subdifferential of ƒ_(i); receiving, by theoptimization layer and from a prior layer of the neural networkarchitecture, input values corresponding to vector c; iterativelycomputing, by the optimization layer, values for vectors c and d todetermine fixed point values c* and d*, wherein each computation of avalue for vectors c and d is of the form:d ^(n) =Hc ^(n) and c ^(n+1)=σ(d ^(n)) where n demotes the n-thiteration of the optimization layer, and c^(n) and d^(n) denote the n-thvalue for vectors c and d; determining fixed point values a* and b*based on applying the inverse of the scattering coordinate transform tofixed point values c* and d*; providing, by the optimization layer,first output based on the determined fixed point values a* and b* to anext layer of the neural network architecture; determining an errorbased on second output of the neural network architecture, wherein thesecond output of the neural network architecture is based on the firstoutput of the optimization layer based on fixed point values a* and b*;backpropagating the determined error through the plurality of layers,wherein the backpropagating comprises determining an updated set ofparameters associated with the convex optimization program based onapplying gradient descent to the linear component H and the non-linearcomponent σ(·); and generating one or more predictions by the neuralnetwork architecture based on a trained set of parameters associatedwith the convex optimization program.
 2. The method of claim 1, whereinthe neural network architecture is configured to generate predictionsregarding speech recognition.
 3. The method of claim 1, wherein theneural network architecture is configured to generate predictionsregarding image recognition.
 4. The method of claim 1, wherein theneural network architecture comprises a convolutional neural network. 5.The method of claim 1, wherein the neural network architecture comprisesa recurrent neural network.
 6. The method of claim 1, wherein the neuralnetwork architecture comprises a feed forward neural network.
 7. Themethod of claim 1, wherein the optimization layer has a direct formstructure, from an input of the optimization layer, consisting of asingle path comprising a single matrix multiplication followed by anon-linearity.
 8. The method of claim 1, wherein the optimization layerhas a residual form structure, from an input of the optimization layer,comprising: a first path, comprising a single matrix multiplicationfollowed by a non-linearity, a second path, comprising an identity pathfrom the input of the optimization layer, and a weighted combination ofthe input and the product of the single matrix multiplication and thenon-linearity.
 9. The method of claim 1, wherein the convex optimizationprogram is a linear optimization program.
 10. The method of claim 1,wherein the convex optimization program is a quadratic optimizationprogram.
 11. A computing device configured to embed a convexoptimization program as an optimization layer in a neural networkarchitecture comprising a plurality of layers, the computing devicecomprising: one or more processors; and memory storing instructionsthat, when executed by the one or more processors, cause the computingdevice to: determine a set of parameters associated with the convexoptimization program, wherein the set of parameters comprises a vectora=(a₁, a₂) of primal decision variables associated with the convexoptimization program and a vector b=(b₁, b₂) of dual decision variablesassociated with the convex optimization program, where:Aa ₁ =a ₂ and b ₁ =−A ^(T) b ₂ and where A is a coefficient matrixcorresponding to one or more constraints of the convex optimizationprogram, and A^(T) denotes the transpose of matrix A; determine a set ofintermediary functionals ƒ_(i)=(ƒ₁, ƒ₂), wherein the value of ƒ_(i) isdefined as equal to a cost term associated with a corresponding a_(i)for a range of permitted values of a_(i) and equal to infinity for arange of unpermitted values of a_(i), wherein the one or moreconstraints of the convex optimization program define the range ofpermitted values and the range of unpermitted values; generate a set ofnetwork variables associated with the optimization layer based onapplying a scattering coordinate transform to vectors a and b, whereinthe set of network variables comprises a vector c=(c₁, c₂) correspondingto input to the optimization layer and a vector d=(d₁, d₂) correspondingto an intermediate value of the optimization layer; generate a linearcomponent H of the optimization layer by applying a first transformationto coefficient matrix A, where the first transformation is of the form:$H = {\begin{bmatrix}I & {- A^{T}} \\A & {I\mspace{34mu}}\end{bmatrix}\begin{bmatrix}{I\mspace{25mu}} & A^{T} \\{- A} & {I\mspace{20mu}}\end{bmatrix}}^{- 1}$ where I is the identity matrix; generate anon-linear component σ(·) of the optimization layer by applying a secondtransformation to the intermediary functionals ƒ_(i), where the secondtransformation is of the form:σ_(i)(d _(i))=(−1)^(i)(∂ƒ_(i) −I)(I+∂ƒ _(i))⁻¹(d _(i)),i=1,2, where∂ƒ_(i) corresponds to the subdifferential of ƒ_(i); generate theoptimization layer as a direct form structure, from an input of theoptimization layer, consisting of a single path comprising a singlematrix multiplication followed by a non-linearity; receive, by theoptimization layer and from a prior layer of the neural networkarchitecture, input values corresponding to vector c; iterativelycompute, by the optimization layer, values for vectors c and d todetermine fixed point values c* and d*, wherein each computation of avalue for vectors c and d is of the form:d ^(n) =Hc ^(n) and c ^(n+1)=σ(d ^(n)) where n demotes the n-thiteration of the optimization layer, and c^(n) and d^(n) denote the n-thvalue for vectors c and d; determine fixed point values a* and b* basedon applying the inverse of the scattering coordinate transform to fixedpoint values c* and d*; provide, by the optimization layer, first outputbased on the determined fixed point values a* and b* to a next layer ofthe neural network architecture; determine an error based on secondoutput of the neural network architecture, wherein the second output ofthe neural network architecture is based on the first output of theoptimization layer based on fixed point values a* and b*; backpropagatethe determined error through the plurality of layers by at leastdetermining an updated set of parameters associated with the convexoptimization program based on applying gradient descent to the linearcomponent H and the non-linear component σ(·); and generating one ormore predictions by the neural network architecture based on a trainedset of parameters associated with the convex optimization program. 12.The computing device of claim 11, wherein the convex optimizationprogram is a linear optimization program.
 13. The computing device ofclaim 11, wherein the convex optimization program is a quadraticoptimization program.
 14. The computing device of claim 11, wherein theneural network architecture comprises a convolutional neural network.15. The computing device of claim 11, wherein the neural networkarchitecture comprises a recurrent neural network.
 16. The computingdevice of claim 11, wherein the neural network architecture comprises afeed forward neural network.
 17. A non-transitory computer readablemedium comprising instructions that, when executed by one or moreprocessors, cause a computing device to perform steps configured toembed a convex optimization program as an optimization layer in a neuralnetwork architecture comprising a plurality of layers, the stepscomprising: determining a set of parameters associated with the convexoptimization program, wherein the set of parameters comprises a vectora=(a₁, a₂) of primal decision variables associated with the convexoptimization program and a vector b=(b₁, b₂) of dual decision variablesassociated with the convex optimization program, where:Aa ₁ =a ₂ and b ₁ =−A ^(T) b ₂ and where A is a coefficient matrixcorresponding to one or more constraints of the convex optimizationprogram, and A^(T) denotes the transpose of matrix A; determining a setof intermediary functionals ƒ_(i)=(ƒ₁, ƒ₂), wherein the value of ƒ_(i)is defined as equal to a cost term associated with a corresponding a_(i)for a range of permitted values of a_(i) and equal to infinity for arange of unpermitted values of a_(i), wherein the one or moreconstraints of the convex optimization program define the range ofpermitted values and the range of unpermitted values; generating a setof network variables associated with the optimization layer based onapplying a scattering coordinate transform to vectors a and b, whereinthe set of network variables comprises a vector c=(c₁, c₂) correspondingto input to the optimization layer and a vector d=(d₁, d₂) correspondingto an intermediate value of the optimization layer; generating a linearcomponent H of the optimization layer by applying a first transformationto coefficient matrix A, where the first transformation is of the form:$H = {\begin{bmatrix}I & {- A^{T}} \\A & {I\mspace{34mu}}\end{bmatrix}\begin{bmatrix}{I\mspace{25mu}} & A^{T} \\{- A} & {I\mspace{20mu}}\end{bmatrix}}^{- 1}$ where I is the identity matrix; generating anon-linear component σ(·) of the optimization layer by applying a secondtransformation to the intermediary functionals ƒ_(i), where the secondtransformation is of the form:σ_(i)(d _(i))=(−1)^(i)(∂ƒ_(i) −I)(I+∂ƒ _(i))⁻¹(d _(i)),i=1,2, where∂ƒ_(i) corresponds to the subdifferential of ƒ_(i); generating theoptimization layer as a residual form structure, from an input of theoptimization layer, comprising: a first path, comprising a single matrixmultiplication followed by a non-linearity, a second path, comprising anidentity path from the input of the optimization layer, and a weightedcombination of the input and the product of the single matrixmultiplication and the non-linearity; receiving, by the optimizationlayer and from a prior layer of the neural network architecture, inputvalues corresponding to vector c; iteratively computing, by theoptimization layer, values for vectors c and d to determine fixed pointvalues c* and d*, wherein each computation of a value for vectors c andd is of the form:d ^(n) =Hc ^(n) and c ^(n+1)=σ(d ^(n)) where n demotes the n-thiteration of the optimization layer, and c^(n) and d^(n) denote the n-thvalue for vectors c and d; determining fixed point values a* and b*based on applying the inverse of the scattering coordinate transform tofixed point values c* and d*; providing, by the optimization layer,first output based on the determined fixed point values a* and b* to anext layer of the neural network architecture; determining an errorbased on second output of the neural network architecture, wherein thesecond output of the neural network architecture is based on the firstoutput of the optimization layer based on fixed point values a* and b*;backpropagating the determined error through the plurality of layers,wherein the backpropagating comprises determining an updated set ofparameters associated with the convex optimization program based onapplying gradient descent to the linear component H and the non-linearcomponent σ(·); and generating one or more predictions by the neuralnetwork architecture based on a trained set of parameters associatedwith the convex optimization program.